Joint Base Station Clustering and Beamforming for Non-Orthogonal Multicast and Unicast Transmission with Backhaul Constraints

The demand for providing multicast services in cellular networks is continuously and fastly increasing. In this work, we propose a non-orthogonal transmission framework based on layered-division multiplexing (LDM) to support multicast and unicast services concurrently in cooperative multi-cell cellular networks with limited backhaul capacity. We adopt a two-layer LDM structure where the first layer is intended for multicast services, the second layer is for unicast services, and the two layers are superposed with different beamformers and decoded by each receiver using successive interference cancellation. We formulate a joint multicast and unicast beamforming problem with adaptive base station (BS) clustering that aims to maximize the weighted sum of the multicast rate and the unicast rate under per-BS power and backhaul constraints. The problem is a non-convex mixed-integer non-linear programming problem. We first develop a branch-and-bound algorithm to find its global optimum. We then reformulate the problem as a sparse beamforming problem and propose a low-complexity algorithm based on convex-concave procedure. Simulation results demonstrate the significant superiority of the proposed LDM-based non-orthogonal scheme over the orthogonal schemes in terms of the achievable multicast-unicast rate region.